Keyword Generation for Biomedical Image Retrieval with Recurrent Neural Networks

This paper presents the modeling approaches performed by the FHDO Biomedical Computer Science Group (BCSG) for the caption prediction task at ImageCLEF 2017. The goal of the caption prediction task is to recreate original image captions by detecting the interplay of present visible elements. A large-scale collection of 164,614 biomedical images, represented as imageID caption pairs, extracted from open access biomedical journal articles (PubMed Central) was distributed for training. The aim of this presented work is the generation of image keywords, which can be substituted as text representation for classifications tasks and image retrieval purposes. Compound figure delimiters were detected and removed as estimated 40% of figures in PubMed Central are compound figures. Text preprocessing such as removal of stopwords, special characters and Porter stemming were applied before training the models. The images are visually represented using a Convolutional Neural Network (CNN) and the Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) Show-and-Tell model is adopted for image caption generation. To improve model performance, a second training phase is initiated where parameters are fine-tuned using the pre-trained deep learning networks Inception-v3 and InceptionResNet-v2. Ten runs representing the different model setups were submitted for evaluation.